Improve admin mapping change output#1206
Open
jwils wants to merge 2 commits into
Open
Conversation
7123aca to
37b0486
Compare
37b0486 to
c6e3e1f
Compare
c6e3e1f to
a9a165f
Compare
a9a165f to
c72908c
Compare
c72908c to
a335f6c
Compare
2e5fc2b to
fcba5e2
Compare
Dropping a field from an index template is a real datastore-state change with operational risk (new rollover indices are auto-created from the template with dynamic: strict mappings), and we previously had a near-SEV from dropping a template field while old indexers still published it. Re-record that history and the required schema evolution workflow at the decision point, since this PR removes the merge_properties guard that previously preserved template fields.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
elasticgraph-adminso all admin users benefit.Behavioral change: index templates now drop removed fields
This is more than an output change. Previously,
merge_propertiespreserved existing template fields forever (a guard added after a near-SEV caused by dropping a template field while deployed indexers still published it). This PR removes that guard: templates are now written to exactly match the schema, so removed fields are dropped and stale fields no longer accumulate.The operational caveat (now documented in
ForIndexTemplate#put_index_template): new rollover indices are auto-created from the template at indexing time withdynamic: strictmappings, so a field must only be removed from the schema once no deployed indexer still publishes it (and no old events containing it will be replayed). Dropped fields appear as deletions in the reported template diff, giving operators a chance to catch a premature removal. Concrete indices are unaffected — their existing fields are always preserved, since the datastore doesn't support removing index mapping fields.Context
Mapping removals are unsupported by Elasticsearch/OpenSearch concrete indices, so showing removed fields every admin run just makes the diffs harder to read. Index templates can be rewritten without those fields, so this keeps existing concrete indices stable while allowing future indices to reflect the current schema. This addresses #766 and the review feedback to improve
elasticgraph-admindirectly instead of adding anelasticgraph-admin_lambdaopt-in.Verification
script/lintscript/spellcheckscript/type_checkscript/quick_build